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ABSTRACT 


Quality assessment is an essential component of education that allows educators to support student 
learning and improve educational programs. The purpose of this study is to evaluate the current state 
of assessment in engineering entrepreneurship education. We identified 52 assessment instruments 
covered in 29 journal articles and conference proceedings that focused on engineering entrepre¬ 
neurship. We evaluated these instruments using the unified theory of validity as a framework. Our 
analysis identified a variety of means through which entrepreneurial knowledge, skills, and attitudes 
are assessed in engineering. Self- or peer-report surveys, some of which were originally developed 
in business contexts, were the primary tool used for assessment. Another common tool was project 
deliverables. The assessment instruments often lacked features that can help differentiate levels of 
competencies and hence had limited utility for formative purposes. We argue that engineering en¬ 
trepreneurship education would benefit from a system of assessment instruments designed through 
rigorous methods and developed to assess constructs specific to entrepreneurial engineering. 

Keywords: Assessment, Entrepreneurial Engineering, literature Synthesis, Entrepreneurship Education 


INTRODUCTION 


Entrepreneurship education, especially with a focus on innovation, is an area of growing need 
and interest within engineering education [1-3], Educators advocate for entrepreneurship education 


WINTER 2016 


1 




ADVANCES IN ENGINEERING EDUCATION 

Evaluation Of Current Assessment Methods In Engineering 

Entrepreneurship Education 


in engineering for diverse reasons. Some argue that preparing students for their careers in industry 
necessitates helping students develop entrepreneurship-related knowledge, skills, and attitudes to 
complement traditional engineering education competencies such as technical problem-solving [1,4,5], 
Entrepreneurial engineers are expected to become leaders at the top of their organizations, to redefine 
markets, and to outperform their competition [6]. Others view entrepreneurial engineers as a driving 
force behind solutions to major technological problems and economic crises at both global and national 
levels [7,8], Moreover, courses and programs with a focus on entrepreneurship can attract a diverse 
and exceptional student population to engineering programs [9,10], improve academic motivation and 
persistence in engineering [11], and inspire students to pursue their own entrepreneurial projects and 
ventures [12], Regardless of differences in motivations, many educators desire entrepreneurial engineers, 
and believe academic courses and programs can help students develop into such engineers [1,6,13-15], 

The trend towards developing new entrepreneurship courses and programs builds upon Standish- 
Kuon and Rice’s [16] argument that entrepreneurial skills can be taught. Yet little research exists on 
the evaluation of student learning. We argue that high quality assessment instruments are needed in 
order to provide valid and reliable evaluation of engineering students’ entrepreneurial development. 
While many programs use assessment instruments, the quality and diversity of these assessment 
instruments need to be determined. 

The purpose of this study is twofold. First, we present the current state of assessment in engi¬ 
neering entrepreneurship education by synthesizing published assessment instruments compiled 
through a comprehensive literature review. This review helps identify the emphasis in literature and 
gaps in assessment. Second, we provide strategies to develop robust practices and research on 
assessment so that the gaps in engineering entrepreneurship can be addressed. 

Need for Research on Assessment 

Assessment has an important role in education. Quality assessment provides valid and reliable 
inferences that serve two critical purposes: to improve student learning (formative) and to improve 
curriculum, instruction, and programs (summative) [17], Whether the purpose is formative or sum- 
mative, the first step in assessment is developing a clear definition and understanding of what 
constitutes engineering entrepreneurship. Which types of knowledge and skills do students need? 
How can students demonstrate their competencies? What attitudes are essential for entrepreneurial 
engineers? The courses and entrepreneurship programs developed for engineering students as well 
as the assessment approaches used in relationship to these efforts provide some, often implicit, 
answers to these questions. Hence, our review provides not only an overview of the current state 
of assessment instruments but also a working definition of engineering entrepreneurship that we 
retrieved by analyzing the knowledge, skills, and attitudes that are emphasized in these efforts. 
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Current Status of Assessment in Entrepreneurial Engineering 

Over the last ten years, three critical studies have evaluated the status of engineering entre¬ 
preneurship assessment [18-20], The results of these studies find a dearth of high quality instru¬ 
ments in entrepreneurship, especially those that are directly relevant to engineering and can be 
used beyond the local classroom context. Duvai-Couetil and her colleagues [19] note that a lack 
of consistency among engineering entrepreneurship courses and programs leads to difficulty 
generalizing assessment instruments to multiple contexts. They also note that many of the as¬ 
sessment instruments come directly from the management field rather than focusing on concepts 
unique to engineering entrepreneurship. Shartrand and her colleagues [20] further note that even 
within engineering-focused entrepreneurship education, the assessment of long-term impact, 
such as a program’s effects on retention and post-graduation success is common, but the short¬ 
term evaluations needed to assess student learning and improve pedagogical practices have yet 
to reach a critical mass [20], 

Among the short-term assessment instruments used in practice, many lack validity and reliability 
evidence necessary to support their usefulness and appropriateness. Besterfield-Sacre and colleagues 
[18], for example, surveyed 126 instructors (93 of which were in engineering disciplines) regarding 
the assessment methods used in their entrepreneurship courses. The results of this survey indicate a 
potentially large quantity and variety of available engineering entrepreneurship assessment instru¬ 
ments, but these assessment instruments are coupled with little evidence related to their effective use 
(e.g., reliability and validity). For example, only 40% of the instructors surveyed described evidence 
of validity. Further, among those instructors describing validity evidence, only 18% indicated validity 
outside their local context (e.g., use in industry or other institutions). It is therefore unclear whether 
these assessment instruments are generalizable to contexts beyond those in which they are employed, 
and whether they address the validity and reliability issues called for by Shartrand and colleagues [20], 

Because assessment development is an on-going and iterative process, research in this area, 
especially in an emerging field like engineering entrepreneurship, is likely to be slow [21], In fact, 
many fields lack sufficient research on assessment despite the critical role and potential benefits 
of assessment with appropriate use [22], The paucity of assessment research and development, 
however, also highlights the opportunities we have to build a strong foundation as a community. 

Framing the Study 

In this study, we use a framework built on recommendations by a recent National Research Coun¬ 
cil (NRC) report on assessment [17] and Messick’s unified theory of validity [23], Our framework 
not only guides the evaluation of the current assessment methods but also provides a structure for 
filling the assessment gap. 
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The NRC report on assessment, led by James Pellegrino, states that assessment is a process of 
reasoning from evidence. The report also highlights that every assessment stands on three pillars: 
how students represent knowledge and develop skills, competencies, and attitudes; tasks or activities 
that allow us to observe these representations; and inferences on student learning that can be derived 
from these observations. Assessment instruments, therefore, should be developed in light of research 
and the best available understanding of how students develop competence in a domain and represent 
knowledge, skills, and attitudes. Only then can we be confident in our inferences on student learning. 

Messick [23, pg. 13] defines validity as “an integrated evaluative judgment of the degree to which em¬ 
pirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and 
actions based on test scores or other modes of measurement.” According to his unified theory, aspects 
of validity previously viewed as separate forms of validity (such as content and criterion-related validity) 
are now considered as different facets of a single construct validity. In addition, Sireci [24, pg. 477] as 
well as Douglas and Purzer [21] outline fundamental aspects of validity that are often misunderstood: 

• Validity is not a property of an assessment instrument; hence, an instrumentcannot be called 
valid. Rather, it refers to the interpretations driven from assessment results and their use for 
a particular purpose. 

• Multiple sources of evidence should be presented to evaluate an assessment method’s utility 
and appropriateness for a specific purpose. 

• If an assessment instrument is to be used for a purpose different than what it is initially designed 
for sufficient evidence must be put forward to defend the use of the test for the new purpose. 

• Validation is not a static, one-time event; but a continuous and iterative process. 

As these researchers highlight, validity should be demonstrated in diverse ways. Empirical evidence 
necessary for establishing validity can take many forms. These can include content analyses through 
reviews by experts, focus group interviews, think aloud protocols with experts and novices, correla¬ 
tion studies showing the assessment’s relationship to another variable identified by theory, factor 
analyses showing structure of the assessment, and pre- and post-test or group comparison studies 
demonstrating an expected difference [25], The quality of an assessment depends on the quality of 
data and evidence derived from these methods. In addition, assessment instruments should have 
instructional sensitivity shown by their alignment with curriculum and instruction [26,27]. In summary, 
validation is accomplished by collecting evidence and making scientifically sound arguments to sup¬ 
port the intended interpretation of assessment scores and their relevance to the proposed use [28], 

While theories on validity are well established, applying theory to practice is a challenging task. 
To support practical application from theory, we translated critical aspects of validity theory to the 
context of engineering entrepreneurship education and developed a four-step process composed 
of a series of key questions that should be asked when developing assessment instruments (See 
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STEP i 

What is the construct to be assessed? What entails entrepreneurial thinking? 

■ What knowledge, skills, and attitudes do student need to learn? 

• How can students demonstrate this learning? 


STEP II 

What purpose does the assessment sen e? 

Evaluate and support student learning (formative)? 

Evaluate and improve curriculum, instruction, programs (summative)? 


STEP III 

How are the assessment data collected? 

What form of assessment would provide strong validity and reliability evidence? 
Would these assessments provide fair and unbiased information on student learning? 
Are the items or protocols developed based on well-established procedures? 


STEP IV 

What is the theoretical rationale and empirical evidence supporting the validity arguments? 

■ Is empirical evidence (statistical or observation methods) used to evaluate the 
appropriateness or structure of the assessment instrument? 


Figure 7. Four-step process for evaluating and designing assessment instruments. 


Figure 1). More importantly, though, this model should help evaluate assessment tools and methods 
allowing educators to arrive at scores that are meaningful, useful, and appropriate for their selected 
purpose. A high quality assessment instrument would: 1) measure a clearly stated construct defined 
by clear and measurable learning objectives, 2) serve a purpose that can be formative or summative 
by design, 3) provide unbiased, reliable information on student learning, and 4) be accompanied 
with evidence supporting validity arguments. 

The model outlined in Figure 1 starts with a first step of defining the construct (knowledge, skills, 
or attitudes) to be assessed. Previous studies and arguments on entrepreneurial thinking in engineer¬ 
ing [29] can inform these decisions. The second step is determining the purpose of an assessment. 
These include formative purposes such as providing students feedback on their learning or summative 
purposes such as program evaluation. Next decision is the use of appropriate assessment instruments 
for evaluating given knowledge, skills, or attitudes. Finally, as a foundation to all previous steps, there 
should be justification that arguments made on student scores provide rationale for their use. 


RESEARCH DESIGN 


To evaluate the status of assessment in entrepreneurial engineering, we performed a search of 
the literature and conducted a content analysis on this literature. The following sections detail the 
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literature search procedure, the sample of peer-reviewed journals and conference proceedings de¬ 
tailing assessment instruments, and our procedure for analyzing these data. 

Selection of Assessment Instruments 

Since the purpose of this study was to determine the current state of assessment in engineering 
entrepreneurship education, we focused our analysis on studies that were conducted in engineering 
education or with engineering students. Our literature search started by accessing four databases 
which include peer-reviewed publications on engineering education and entrepreneurship studies: 
ABI Inform/Complete, Business Source Premier, Compendex, and Web of Knowledge. We searched 
abstracts in these databases for the terms “engineer*”; “entrepreneur*”; and at least one of “assess*”, 
“measure*”, and “metric”. This search resulted in 396 journal articles and peer-reviewed conference 
publications. We reduced the sample to 105 articles by reviewing the content of each article’s ab¬ 
stract and removing articles that did not describe an assessment instrument or duplicated previ¬ 
ous instruments. We then viewed the text of each article and evaluated whether the assessment 
instruments therein were described in sufficient detail and evaluated entrepreneurial development 
among engineering students. In the event that a paper described a previously developed assess¬ 
ment instrument, we identified the paper in which the assessment instrument was initially described 
and added it to the sample. 

The literature search resulted in a sample of 29 articles that described 52 unique instruments 
or metrics to assess engineering entrepreneurship. The Appendix represents the article database. 
Although our search was extensive, a majority of the articles that made up our pool were confer¬ 
ence proceedings from key engineering education conferences (59% of the sample). Twelve articles 
(41% of the sample) were published in journals that were equally distributed between business and 
engineering. 

Data Analysis 

We performed a content analysis of the final 29 articles and 52 assessment instruments with a 
goal of identifying key components of entrepreneurial activity in engineering classrooms and the 
validity of assessment in those contexts. 

Content Analysis Coding Protocol 

While the unified theory of validity guided the structure, Besterfield-Sacre and colleagues’ [18] 
descriptive study of the types of assessment instruments used by entrepreneurship instructors 
provided an initial classification scheme. This classification scheme contained five key categories: 
type of assessment instrument/method-, type of knowledge, skill, or attitude measured ; (when) 
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Category 

Examples 

Topic assessed 

Leadership, risk-taking, communication 

Dimension of topic assessed 

Knowledge, skill, attitude 

Purpose of assessment 

Formative feedback, summative evaluation 

Individuals assessed 

Engineering students, capstone design teams, seniors in mechanical engineering 

Type of assessment 

Survey, business proposal, prototype, quiz 

Evaluators/assessors 

Faculty, student peers, self, industry professional 

Reliability/validity evidence 

Internal consistency, inter-rater reliability, group comparison 

Table 7. Categories for Initial Coding Phase. 


instrument/method was used ; (population) instrument/method used to evaluate ; and focus of the 
instrument (e.g., process or product). The scheme captured evidence on the instruments’ use beyond 
the setting in which it was initially developed and employed. 

We expanded the original classification scheme to include categories germane to our study (See 
Table 1). The dimension of topic assessed category was added to provide a more finely grained 
analysis of the topics assessed. For example, assessment instruments that evaluate attitudes towards 
business planning provide different information than assessment instruments that evaluate business 
planning skills such as need identification. The purpose of assessment category was added due to 
the importance of determining whether the assessment instrument was used for formative feedback 
or summative evaluation [17], Further, the validity category was modified to include reliability evi¬ 
dence. We also excluded or modified categories such as the when assessment is used category as 
it had more to do with classroom logistics than content and strength of the assessment instrument. 

During an initial coding phase, two researchers reviewed a subset of twenty assessment instru¬ 
ments in light of the initial coding categories described in Table 1. The researchers suggested new 
and revised codes whenever an aspect of an assessment instrument in the sample was not covered 
by the then-current coding scheme. The two researchers repeated this process iteratively, adding 
additional assessment instruments with each iteration, until they agreed on a final coding scheme 
that represented the extent of assessments reviewed. This final coding scheme is represented in 
Table 2. 

The overall category structure of this coding scheme was different than the initial protocol in two 
ways. First, we removed the individuals assessed category. This category was unnecessary since 
each of the assessment instruments we identified was given to engineering students and a more 
finely grained analysis of assessment populations was beyond the scope of this study. Second, we 
added an assessment context category. This category emerged from the initial validity category 
(i.e., whether the instrument was used in other settings) as our new validity/reliability category 
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Category 

Codes 

Assessment context 

General, Local 

Assessment purpose 

Formative, Summative 

Assessment type 

Concept Map, Essay, Interview, Observation, Project Deliverable, Quiz, Survey 

Topic assessed 

Business Planning, Business Realization, Communication, Design, Entrepreneurship 
General, Leadership, Professional Practice, Teamwork 

Topic dimension assessed 

Knowledge, Skill, Attitude 

Validity evidence 

Face Validity, Content Validity, Construct Validity, Reliability 


Table 2. Final Coding Scheme. 


focused on additional measures of reliability (such as internal consistency and inter-rater reliability) 
and validity (such as group comparison and expert review). Identifying the context for which the 
assessment was developed is important for determining the extent to which these instruments are 
disseminated and the ability to adapt their use to additional settings. 

Once the coding protocol was established, one researcher reviewed the entire set of assessments 
and classified each assessment using the final coding scheme. A second researcher reviewed a sub¬ 
sample of 13 instruments (25% of the total sample), at an initial agreement level of 65%. The two 
researchers then met to discuss disagreements and clarify understanding of the coding scheme. 
A majority of the disagreements pertained to assessment instruments that registered multiple 
codes in one category. For example, one researcher initially coded “entreprepeneurship general” 
on instruments for which the other researcher coded multiple topics assessed. Once these issues 
were identified, the researchers reached 100% agreement on a sub-sample of 13 assessment instru¬ 
ments. The initial coder then recoded the remaining data based on the refined understanding of 
the coding scheme. 


RESULTS 


What is the Construct to Be Assessed? What Entails Entrepreneurial Knowledge, Skills, or 
Attitudesin Engineering? 

In everyday language, and often in research literature, terms such as innovator, inventor, and en¬ 
trepreneur are used interchangeably. According to theories of validity, the construct to be assessed 
should first be clearly defined and delimited [30], Hence, before we attempt to assess student’ 
competencies related to engineering entrepreneurship, we must explicitly define what construct or 
constructs comprise engineering entrepreneurship. Common methods used to establish construct 
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definitions include expert consultations, interviews, and reviews of prior studies [31]. Here, we pres¬ 
ent patterns of agreement in defining engineering entrepreneurship retrieved from our review of 
the literature on assessment and curriculum development. 

Definitions of the Entrepreneurial Engineer and Engineering Entrepreneurship 

The construct of engineering entrepreneurship is discussed along three dimensions in the publi¬ 
cations we reviewed: knowledge, skills, and attitudes (i.e., mindset). Knowledge refers to concepts 
students need to know and understand about engineering entrepreneurship. While entrepreneurial 
engineers are viewed as those who understand and promote entrepreneurship within companies 
[6], they are expected to have the knowledge of typical engineers as well as an understanding of 
economic and financial concepts [1,32]. More specifically, engineering entrepreneurs must demon¬ 
strate the math, science, and technical knowledge necessary to develop solutions to engineering 
and technical problems as well as knowledge of startup capital for new ventures, market forces, 
sales, intellectual property, and finance [7,29], 

An engineering entrepreneur is also defined as one who organizes and manages the risk of an 
engineering business or enterprise [36]. They are required to have the skills to carry out or promote 
entrepreneurial ventures [35], Noted skills of entrepreneurial engineers include the ability to cre¬ 
ate unique solutions to current problems [37], ability to create and carry out successful business 
ventures such as writing effective business plans [35], and ability to organize and manage projects 
and businesses [4,38], 

The Kern Foundation’s Engineering Entrepreneurship Network (KEEN) emphasizes the impor¬ 
tance of mindset, stating that entrepreneurial^ minded engineers appreciate societal values of 
products they create and persist in an orientation towards customer needs [6]. Entrepreneurial^ 
minded engineers also have high entrepreneurial self-efficacy and show tendency towards risk¬ 
taking, persistence, autonomy, achievement, and leadership [33,34], Entrepreneurial engineers are 
also called upon to demonstrate more general professional skills such as leadership, teamwork, and 
communication in addition to traditional technical skills [1,2,35], 

Based on the literature then, a working definition of engineering entrepreneurship would be: The 
knowledge, skills, and attitudes necessary to envision, lead, communicate, develop, and realize new 
technological and socia/ly-sensitive processes and products in an established corporate setting or 
as part of a new venture. 

Frequency of Assessment Topics 

The 52 assessment instruments demonstrated a broad range of topics applicable to engineering entre¬ 
preneurship that were consistent with the above literature. These topics, described in Table 3, represent 
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Category 

Topic 

Description 

Business 

Aspect 

Business 

Planning 

Knowledge, skills, and attitudes necessary to begin new business ventures or projects, 
including identifying need areas and developing business plans. 


Business 

Realization 

knowledge, skills, and attitudes necessary to promote the development of solutions 
and bring those solutions to market, particularly relating to core business concepts 
such as budgeting, finance, marketing, and human resources. 

Engineering 

Aspect 

Design 

Ability to conceptualize and develop new and better solutions based on existing need 
areas. 

Professional 

Aspect 

Communication 

Ability to communicate ideas and plans both in writing and orally through papers, 
presentations, and design/project reviews 


Leadership 

Knowledge, skills, and attitudesrelated to setting the direction or scope of team 
projects or organizing/uniting a disparate team of people to work together effectively. 


Professional 

Practice 

Awareness of and ability to deal with situations that may involve unethical practices 
or those that challenge some professional code or standard of conduct 


Teamwork 

The ability of individuals or groups to function in or as a team. Knowledgeskills, and 
attitudesnecessary to function effectively. 

General 

Entrepreneurship 

General 

A broad conceptualization of entrepreneurship without reference to any specific 
knowledgeskills, or attitudes. 


Table 3. Entrepreneurship Topics Assessed in Literature. 
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Figure 2. Frequency of Topics Assessed. 
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business-related aspects such as business planning, professional aspects such as communication, and 
engineering aspects such as design. As shown in Figure 2, business planning was the most frequently 
assessed topic (48% of assessment instruments). Other key topics were design (42%), communication 
(38%), business realization (33%), teamwork (33%), and leadership (27%). An interesting omission is 
consideration of technical engineering knowledge. While previous literature suggests that engineering 
entrepreneurs must have relevant technical knowledge in addition to business and social awareness, 
outside of design, the topics assessed align with a more general conceptions of entrepreneurship. 
There was no clear evidence that technical competencies were integrated with business competen¬ 
cies, although it is possible that technical competencies were assessed using different instruments. 

Figure 3 presents the frequency with which each dimension of engineering entrepreneurship was 
assessed in the reviewed literature. It should be noted that some assessment instruments focused 
on multiple dimensions, thus values in Figure 3 will sum to greater than 52. The majority (65%) of 
the instruments focused on skill assessment. Only 29% of instruments assessed attitudes such as 
self-efficacy beliefs or entrepreneurial mindsets, and 21% assessed entrepreneurial knowledge. These 
differences are perhaps a result of the focus on project deliverables and self-report surveys, which 
tended to favor skill assessment. Previous literature demonstrates that self-efficacy, risk-taking, and 
customer orientation and knowledge of economics are also critical [19,33,34], thus the infrequency 
of knowledge and attitude assessments represents a gap in the literature. 

Our analysis also included a more finely grained evaluation of the topics assessed within each of 
the three dimensions of assessment. Table 4 shows the joint occurrences of assessment topics and 
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Skill 

Attitude 

Knowledge 

Business Planning 

11 

11 

8 

Business Realization 

8 

4 

7 

Design 

14 

7 

3 

Communication 

18 

3 

2 

Leadership 

9 

6 

4 

Professional Practice 

2 

1 

2 

Teamwork 

14 

4 

2 

Entrepreneurship General 

0 

2 

0 


Table 4. Distribution of entrepreneurship topics by assessment areas. 


dimensions. It is important to note that some of the assessment instruments evaluated multiple top¬ 
ics, as well as topics across multiple dimensions (e.g., knowledge and skills), thus summing across 
rows and down columns provide larger values than suggested by Figures 2 and 3 respectively. The 
publications we reviewed covered a variety of topics ranging from design to leadership to business 
realization, demonstrating a complex categorization of the engineering entrepreneur (Figure 2). 
Table 4 demonstrates that the topics were assessed across the range of topic dimensions, but certain 
key topics were limited in at least one dimension. Design and teamwork, for example, were rarely 
assessed as knowledge. Communication was frequently assessed as a skill. The business-related 
aspects, however, were assessed almost uniformly as knowledge, skills, and attitudes. Since design, 
teamwork, and communication are more broadly applicable to engineering, the unequal distribution 
of assessments may reflect a unique emphasis on skills in these areas within engineering entrepre¬ 
neurship. The focus on skills in those areas may demonstrate the importance of those skills, but may 
also represent an assessment gap to be considered in future assessment development. 

What Purpose does the Assessment Serve? 

Identifying the assessment purpose is an important step because an assessment instrument 
designed for a specific purpose or context may not be appropriate for use in a different context 
[17], We found two dimensions of purpose used for entrepreneurship assessment (see Table 5). The 
first dimension, context of assessment, indicated whether an assessment was designed to be used 
in a single course or program (local context) or in a variety of contexts (general context). While 
assessment instruments designed for local contexts evaluated more precise constructs within lo¬ 
cal environments, assessment instruments for general constructs were more carefully developed 
and coupled with more reliability and validity evidence. The second dimension, feedback structure, 
indicated whether an assessment instrument was used to provide summative feedback, as in the 
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Local Context 

General Context 

Total 

Formative 

5 

none 

5 

Summative 

26 

13 

39 

Formative & Summative 

8 

none 

8 

Total 

39 

13 

52 


Table 5. Distribution of Local and General with Summative and Formative elements. 


overall effect of an entrepreneurship program [39], or for the purpose of formative feedback to 
support student learning [40], 

There are important differences between assessment instruments designed for formative or 
summative purposes. The former approach requires the development of instruments that would 
provide information on students’ progressions in their competencies. These should go beyond 
pre-post assessment and employ milestones showing progressions of individuals or groups of 
students [17], Summative assessment requires an evaluation of final outcomes or change, perhaps 
through pre-post assessment instruments. The key in designing these assessment instruments is 
ensuring the alignment of the assessment with the curriculum and instruction and an awareness 
of levels of transfer required between what is learned and what is expected to be demonstrated 
in an assessment [41], 

The majority of assessment instruments were developed for local use and the most common 
purpose of these assessment instruments was summative evaluation. Forty-seven of the 52 assess¬ 
ment instruments (90%) were designed and used to evaluate student progress, individual courses, 
programs, and minors. Thirty-four of the forty-seven summative assessment instruments (72%) 
were developed in the local settings and were not intended for general distribution. Among the 13 
summative assessment instruments developed for general contexts, seven were developed outside 
of an explicitly engineering context. Only 13 of 52 assessment instruments (26%) were designed 
for a formative purpose. This number includes eight instruments that provided feedback to both 
students and program administrators. 

How are the Assessment Data Collected? 

The analysis resulted in seven distinct types of assessment instruments used in both course and 
general contexts. Table 6 provides an overview of each of the seven types of assessment, their 
strengths and weaknesses. The frequency counts for each type of assessment are also presented in 
Figure 4. The two most popular assessment types were surveys—which included self or peer assess¬ 
ment of entrepreneurial skills, attitudes, and knowledge—and project deliverables—which included 
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Survey Project Essay Quiz Interview Concept Observation 
Deliverable Map 

Figure 4. Frequency Counts of Assessment Types. 


instructor or expert evaluation of student produced project reports, business plans, presentations, 
or design concepts/prototypes. 

Surveys were the most common form of assessment (25 out of 52). Twenty-one of these were 
student self-report surveys. Some asked students to directly evaluate their own capabilities (e.g. 
[11]). Others asked students to group individual items into factors [19], While these surveys are 
straightforward to develop, administer, and analyze, they are prone to self-report bias (i.e. students 
are not always accurate in assessing themselves). This effect seems to be stronger for constructs 
that are difficult to define or identify [42], Peer evaluation surveys, which are also used for entre¬ 
preneurship skill assessment, are prone to similar biases [43,44], 

Project deliverables were the second most common form of assessment (13 of 52 assessment 
instruments). These included reports or project artifacts—such as oral presentations, written re¬ 
ports, design prototypes or conceptual drawings, and written business plans—that were completed 
in fulfillment of a course project. Typically these were assessed by course instructors, members of 
industry, or combinations of the two. Assessment was occasionally guided by a rubric, but was often 
left to the judgment of the assessors. While project deliverables can represent greater authenticity 
they are often difficult to generalize to other contexts due to resources, assignment purpose, and 
subjective scoring. 
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Assessment 

Method 

Strengths 

Weaknesses 

Best Suited 
to Evaluate 

Used in Practice 
to Evaluate 

Survey 

• Validity evidence if provided 

• Straightforward to evaluate 

• Short administration time 

• Self-report or teammate bias 

• Overused for assessing skills 

• Attitudes 

• Knowledge 

• Skills 

• Attitudes 

Project 

Deliverable 

• Classroom-embedded 

• Relies on expert judgment 

• Potential for assessor bias 

• Skills 

• Skills 

Essay 

• Administration time can be 
short 

• Can produce detailed, open- 
ended responses 

• Can be based on or emulate 
authentic situations 

• Evalaution can be labor 
intensive 

• Writing skills may obscure 
content 

• Knowledge 

• Skills 

• Attitudes 

• Knowledge 

• Skills 

• Attitudes 

Quiz 

• Short administration time 

• Easy to assess, can provide 
timely & specific feedback 

• None were developed to 
measure competency 

• Typically designed for local 
contexts 

• Knowledge 

• Knowledge 

Interview 

• Allows depth and flexibility of 
responses 

• Labor intensive 

• Time intensive” 

• Knowledge 

• Attitudes 

• Attitudes 

Concept 

Map 

• Short implementation time 

• Evaluates connections and deep 
understanding 

• Evaluation can be labor 
intensive 

• Students may be unfamiliar 
with method 

• Knowledge 

• Attitudes 

• Knowledge 

Observation 

• Authentic context 

• Evaluation can be labor 
intensive 

• Skills 

• Skills 


Table 6. Engineering Entrepreneurship Assessment Methods. 


Each tool and method has its own strengths and weaknesses. Table 6 provides the authors’ 
comparison of these approaches, which is informed by prior work of Cross and Angelo [45] and 
Palombaand Banta [46] on various forms of assessment. 

What is The Theoretical and Empirical Evidence Supporting the Validity Arguments? 

The weakest aspect of a majority of assessment instruments was related to a lack of an explicit 
theoretical framework or research-based argument that guided their design. Many of the assess¬ 
ment instruments were developed to assess student learning in individual programs or courses 
and needed only be applied in local settings. As a result, assessment content was often guided by 
specific program or course needs. While it is possible that instruments and items therein were de¬ 
veloped with theoretical frameworks or research-based arguments in mind, these were not evident 
in the written reports. 

A handful of assessment instruments borrowed items from research-based general entrepreneur- 
ship assessment instruments such as Chen’s Entrepreneurship Self-Efficacy instrument [47], This 
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instrument measures self-efficacy in entrepreneurial activities such as confidence in new venturing 
and new ideas, risk taking, and performing financial analysis [47, pg. 305], 

It is crucial to identify elements of these more generalized surveys that are applicable to engineering 
entrepreneurship. Further, reliability and validity must be ensured for the modified survey as a whole 
and within the engineering student population. 

Other researchers developed new surveys specifically to assess engineering entrepreneurship. 
Duval-Couetil and her colleagues developed an instrument, the Engineering Entrepreneurship 
Survey, using social cognitive theory [48] as a framework [1], This instrument combined three 
formats including a five-point Likert scale self-assessment instrument of entrepreneurial knowl¬ 
edge, skills, attitudes, and awareness; a dichotomous (yes/no) self-assessment of entrepreneurial 
behaviors, and a ten-point measure of self-efficacy in entrepreneurial activities such as leading 
project teams and estimating project costs. Another survey, the Entrepreneurship Knowledge 
Inventory is based on extensive review of general and engineering entrepreneurship literature 
[7,20], In this instrument, students are asked to assess their level of familiarity with a series 
of terms related to engineering entrepreneurship on a five-point scale as well as their level of 
participation in a series of entrepreneurship-related academic activities on a four-point scale. 
Further, the Engineering Mindset Rubric was developed based upon the framework presented in 
Covin and Slevin's [49] Entrepreneurship Orientation Scale. This instrument categorizes entre¬ 
preneurship along the themes of product market innovation, proactiveness of decision-making, 
and risk taking [20]. 

Only 10 of the 52 assessment instruments (19%) were coupled with direct evidence of validity 
(See Table 7). Among these, six were presented in papers focused on the development and valida¬ 
tion of an assessment of engineering entrepreneurship [1,2,7,20,50,51], An additional four of the 52 
assessment instruments were borrowed or modified from instruments developed and validated in 
general business and entrepreneurship contexts [47,52-54], 

The implementations of these ten assessment instruments in engineering education con¬ 
texts show three key features of validation. First, validity can be demonstrated through a 
variety of means. Techniques used in these implementations include internal consistency, 
inter-rater reliability, group comparison, literature review, iterative development, participant 
review, expert review, and structural equation modeling. Validity contains multiple facets and 
thus a variety of techniques appropriate for the individual context should be utilized. Sec¬ 
ond, validation is an ongoing process. Validation techniques such as expert review, internal 
consistency, and group comparison not only demonstrate accurate and precise assessment 
of key constructs, but provide feedback for future improvements to the instruments. Finally, 
validity in one context does not guarantee validity in another context. The General Enter- 
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Assessment 

Name 

Authors 

Applications in 
Engineering Context 

Reliability Evidence 

Validity Evidence 

Engineering 
Entrepreneurship 
Survey (EES) 

Duval-Couetil, 
Reed-Rhoads, 
Haghighi (2012) 

Concurrent 

implementation at three 
different schools by 
authors 

Internal consistency: 
Cronbach’s a > .80 for all but 
one factor (.74 for other) 

Expert review, 
group comparison, 
literature review, 
participant review 

Entrepreneurial 

Intention 

Questionnaire 

Linan and Chen 
(2009) 

Concurrent 

implementations at two 
schools (See [55]) 

Internal consistency: 
Cronbach’s a > .70 in 
engineering implementation 

Literature review, 
structural equation 
modeling 

Entrepreneurship 
Knowledge 
Inventory (EKI)* 

Besterfield-Sacre, 
Robinson, Shuman, 
Shartrand, and 
Weilerstein (2012) 

Concurrent 
implementation at 10 
different schools by 
authors 

Internal consistency: 
Cronbach’s a > .90 for all 
five factors 

Expert review, group 
comparison 

Entrepreneurship 
Mindset Rubric 

Shartrand, 
Weilerstein, 
Besterfield-S acre, 
Olds(2008) 

Concurrent 

implementation at four 
different schools by 
authors 

Inter-rater reliability: 
Cronbachs’s a = .80 

Expert review 

Entrepreneurship 

Self-Efficacy 

Chen, Greene, and 
Crick (1998) 

Three separate 
implementations at 
different schools (See 
[34,56,57]) 

Internal consistency 
(Cronbach’s a = .92 in single 
engineering implementation; 
Cronbach’s a = .89, .65-.86 on 
individual factors in original 

Group comparison 
(in original paper) 

General 

Enterprising 

Tendency 

Caird (1991) 

Two separate 
implementations at 
different schools (See 
[34,58]) 

Internal consistency: 
Cronbach’s a = .42 overall in 
engineering implementation 

Group comparison 

KEEN - TTI 

Performance 

DNA 

Pistrui, Layer, and 
Dietrich (2012) 

Concurrent 
implementation at 

17 different schools 
and to 313 industry 
professionals by authors 

Internal consistency: 

Cronbach a > .70 

Structural 
equation modeling 
— statistically 
significant path 
coefficient 

Leadership 
Attitudes and 
Beliefs Scale 

Wielkiewicz 

(2000) 

Two separate 
implementations at 
different schools (See 
[34,56]) 

Internal consistency: 
(Cronbach’s a = .82 and 
.80 in separate engineering 
implementations 

Group comparison 

Leadership 

Self-Perception 

Instrument 

Gerhardt, 

Carpenter, Grunow, 
Hayes (2010) 

One implementation by 
authors 

Internal consistency: 
Cronbach’s a > .60 for 74.3% 
of items; Cronbach’s a < .70 
for 9.7% of items 

Based on established 
survey questions, 
participant review 

Survey of 

Entrepreneurial 

Intentions 

Kuckertz and 
Wagner (2010) 

One implementation by 
authors 

Internal consistency: 
Cronbach’s a > .64 for all 
factors 

Group comparison, 
iterative development, 
literature review 

*Previously known 

as the NCIIA Entrepreneurship Inventory 




Table 7. Assessment instruments with Reliability and Validity Evidence. 


prising Tendency instrument, for example, did not translate well to an engineering education 
context as the instrument demonstrated poor internal consistency. Hence with each new 
implementation careful consideration of contextual differences and reevaluation of validity 
evidence are necessary. 
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DISCUSSION AND STRATEGIES FOR CLOSING THE ASSESSMENT GAP 


Our analysis of the literature on engineering entrepreneurship resulted in two key findings. 
First, in agreement with previous similar analyses [18], we found a variety of purposes, methods, 
and forms used to assess engineering entrepreneurship knowledge, skills, and attitudes. However, 
these assessment instruments were not organized in a way that would provide systematic evalua¬ 
tion of curriculum and entrepreneurship programs. Nor did they completely align with conceptions 
of engineering entrepreneurship from the literature. Our analysis provides a structure and a set of 
taxonomies highlighting areas where there is significant emphasis and more work necessary. Po¬ 
tential gaps include limited emphasis on the assessment of knowledge, formative assessment, and 
novel forms of instruments that move beyond surveys or project deliverables. 

Second, the quality of the assessment instruments varied. Two common issues were minimal 
description of instrument development processes and lack of evidence to support the validity of 
inferences derived from these instruments. Moreover, the publications we identified did not always 
differentiate engineering entrepreneurship from a general conception of entrepreneurship. While 
many professional and business-related competencies are considered critical for entrepreneurial 
engineers, it is often not clear what aspects of these competencies are unique to an entrepreneurial 
engineering endeavor. Such understanding is key, as we cannot assume that effective use in man¬ 
agement or business contexts can translate well to engineering education contexts. Conversely, 
assessments often did not focus on technical competencies and thus do not demonstrate how these 
competencies can or should be integrated into entrepreneurial work. In light of these findings, we 
developed a set of recommendations and strategies on how to address them. 

Developing a System of Entrepreneurship Assessment Instruments 

As shown in our study and the previous literature, engineering entrepreneurship requires a vari¬ 
ety of knowledge, skills, and attitudes; thus, it requires a variety of assessment techniques and foci 
[15,38,57,59], As such, multiple forms of assessment instruments are needed to create a complete 
picture of one’s entrepreneurial mindset and competencies [40,59], It is unlikely that a single assess¬ 
ment can comprehensively capture one’s engineering entrepreneurial abilities. Within the 29 papers we 
reviewed, we identified 52 unique assessment instruments. Engineering education would benefit from 
a system of instruments from which educators can select and to which researchers and educators can 
contribute new and revised instruments. This paper identified key gaps such as lack of knowledge and 
attitude assessments, especially in key topic areas such as design and communication; over-reliance 
on self-report surveys and project deliverables; and emphasis on summative rather than formative 
assessment. A first step in establishing the proposed system would be to explore the appropriateness 
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of novel or underutilized assessment techniques and assessment instruments focusing on dimension- 
topic combinations (such as design knowledge and attitudes). Then assessment instruments to ad¬ 
dress gaps deemed as critical could be identified. In addition to addressing these gaps, researchers 
can collectively engage in research in effective use of assessment in diverse educational settings. 

While identifying the variety of assessment instruments across the categories identified in this 
study may guide a comprehensive view of engineering entrepreneurship, it is important not to 
stress haphazard combinations of assessment instruments. Systems can easily become cumber¬ 
some, requiring a significant amount of time for students to complete and educators to analyze, and 
may include redundant or conflicting elements. Hence, a system of assessment instruments would 
need to be effective, complementary, and efficient and provide syntheses of existing elements. The 
design of an online database can facilitate these criteria as well as easy sharing and dissemination 
of assessment instruments [18], 

Identifying Empirical Validity Evidence. Few publications presented validity evidence and thus it 
is difficult to determine whether they consistently measured the intended constructs. For example, 
only ten of the 52 assessment instruments included appropriate validity and reliability data. Valida¬ 
tion studies can show that assessment instruments may vary in what they measure despite the initial 
conceptualization of the educator. For example, McGourty and his colleagues [40] developed and 
compared data from a series of assessment instruments. One of their findings was that conceptual 
understanding of the project realization process, assessed using a concept mapping task, did not 
match the understanding students demonstrated through other course projects. 

In addition, the various ways in which engineering entrepreneurship is assessed show that en¬ 
gineering entrepreneurship is not viewed as a unified construct but composed of diverse knowl¬ 
edge, skills and attitudes. While we propose the need for a system of assessment instruments that 
together form a full definition of entrepreneurial competencies, we also argue for narrower (more 
focused) definitions and assessment instruments. For example, rather than designing an instrument 
on general communication skills, educators should identify and assess communication skills critical 
to engineering entrepreneurship. Effective assessment systems can only be built upon clear defini¬ 
tions that impart specific competencies. The KEEN - TTI Performance DNA is perhaps a step in this 
direction [29,59], Assessment instruments that can provide evidence of validity and reliability are 
critical for the evaluation of student learning and or programs. 

Utilizing Research & Theoretical Frameworks 

Efforts to improve assessment would not be successful unless accompanied with research on 
student learning. Assessment instruments, especially those that can provide formative feedback, 
should be developed in light of studies that can help educators differentiate levels of competencies 
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in engineering entrepreneurship. While limited, there are examples of such research. For example, 
Golish and colleagues [60] compared the project realization processes of industrial and academic 
innovators using a process mapping task. They found that industrial innovators placed stronger 
emphasis on creating customer value and market analysis than academic innovators. Academic 
innovators were more focused on feasibility analysis than industrial innovators. This finding sug¬ 
gests there are different approaches to innovation. Surveys that can measure students’ tendencies 
towards an approach such as valuing customer needs (desirability), market opportunities (viability), 
or technical aspects (feasibility) can provide information on changes in their views. 

Further, researchers might explore methods to classify individuals as novices, emerging experts, 
and experts in engineering entrepreneurship [61], Research using think-aloud protocols can guide 
design of new assessment instruments [62], Other research methods, such as concept map activities, 
observations of student teams, and case study responses present promising opportunities. 

Engineering educators are invited to utilize the assessment framework in Figure 1 to guide the 
evaluation of existing assessment instruments or the design of new ones. This evaluation structure 
and subsequent analysis on the current literature helps provide an overview of current instruments 
and will help strengthen future work in entrepreneurship assessment in engineering. 
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